Vascular data processing and image registration systems, methods, and apparatuses

ABSTRACT

In part, the invention relates to processing, tracking and registering angiography images and elements in such images relative to images from an intravascular imaging modality such as, for example, optical coherence tomography (OCT). Registration between such imaging modalities is facilitated by tracking of a marker of the intravascular imaging probe performed on the angiography images obtained during a pullback. Further, detecting and tracking vessel centerlines is used to perform a continuous registration between OCT and angiography images in one embodiment.

FIELD OF THE INVENTION

In part, the invention relates generally to the field of vascular systemand peripheral vascular system imaging and data collection.

BACKGROUND OF THE INVENTION

Interventional cardiologists incorporate a variety of diagnostic toolsduring catheterization procedures in order to plan, guide, and assesstherapies. Fluoroscopy is generally used to perform angiographic imagingof blood vessels. In turn, such blood vessel imaging is used byphysicians to diagnose, locate and treat blood vessel disease duringinterventions such as bypass surgery or stent placement. Intravascularimaging technologies such as optical coherence tomography (OCT) andacoustic technologies such as intravascular ultrasound (IVUS) and othersare also valuable tools that can be used in lieu of or in combinationwith fluoroscopy to obtain high-resolution data regarding the conditionof the blood vessels for a given subject.

Fractional flow reserve (FFR) can also be used to evaluate a bloodvessel during imaging and angiography. Intravascular OCT, IVUS, and FFRare invasive catheter-based systems that collect optical, ultrasound,and pressure data, respectively, from inside blood vessels or withrespect to a sample of interest. Angiography is a noninvasive x-rayimaging method that collects data from outside the body during injectionof a radio-opaque contrast fluid.

Intravascular optical coherence tomography is a catheter-based imagingmodality that uses light to peer into coronary artery walls and generateimages thereof for study. Utilizing coherent light, interferometry, andmicro-optics, OCT can provide video-rate in-vivo tomography within adiseased vessel with micrometer level resolution. Viewing subsurfacestructures with high resolution using fiber-optic probes makes OCTespecially useful for minimally invasive imaging of internal tissues andorgans. This level of detail made possible with OCT allows a clinicianto diagnose as well as monitor the progression of coronary arterydisease.

Given the complexity of the various technologies described above and theassociated complexity of the datasets each of them generate, performingco-registration between two image-based technologies such as OCT andangiography is time consuming. As a result, challenges regarding realtime co-registration of intravascular image data and angiography imagedata remain. Some co-registration techniques depend heavily on userinteraction. Unfortunately, taxing an operator with significant userinteraction during co-registrations such as requiring manually matchingcorresponding points in images, a long waiting period for the algorithmsto return a co-registration, and finally verifying the results, makessuch approaches impractical in many clinical scenarios. In addition,other approaches use data from asynchronous or third party controlledsources which results in timing irregularities. In addition, sincecontrast agents, such as dyes, are used with some intravascular imagingmodalities that interfere with other noninvasive imaging modalities,imaging artifacts and errors can result which interfere withco-registration between such modalities.

Accordingly, a need therefore exists to address one or more of thechallenges identified above relating to intravascular imaging andangiography imaging. Embodiments of the invention address thesechallenges and others.

SUMMARY OF THE INVENTION

One embodiment of the invention relates to methods for registrationbetween two imaging modalities such as angiography and OCT. Oneembodiment of the invention relates to one or more methods forperforming co-registration between angiography images and the OCTimages.

One embodiment of the invention relates to a method for performingdetection of stationary marker band on a frame without a contrast agentsuch as a dye and with a contrast agent. In addition, one embodiment ofthe invention further provides for tracking of such a marker band as itmoves through a lumen of a blood vessel such that it is tracked onsubsequent pullback frames, including tracking from a frame withoutcontrast agent to a frame with contrast agent.

In one embodiment, the time period to register between about 20 and 100frames of angiography image frames and between about 100 and about 1500frames of OCT image frames ranges from about 2 seconds to about 30seconds. In one embodiment, registration of angiography image data andOCT image data obtained during an OCT pullback are co-registered in lessthan about 10 seconds. In one embodiment, the pullback of a datacollection probe ranges from about 4 to about 10 seconds. In oneembodiment, frames of angiography are obtained in real time using aframe grabber. The frames of angiography data are grabbed in asynchronized manner with the OCT image data frames obtained as a resultof the pullback.

In one embodiment, a co-registration method co-registers an OCT framesof image data obtained during the imaging of a pullback with frames ofangiography data obtained during such a pullback within a registrationtime period of about 3 to about 5 seconds.

In one embodiment, the invention relates to an image data processingsystem that includes a frame grabber, an OCT system configured toperform imaging during pullback of a data collection probe having amarker through a blood vessel and generate time stamped OCT image datawith respect to the blood vessel, one or more computing devices, and auser interface, wherein the frame grabber is configured to obtain timestamped frames of angiography image data with respect to the bloodvessel.

In one embodiment, video capture of angiography image data occurs on theOCT system. In one embodiment, a user manually designates a marker bandon an angiography image. In one embodiment, the designated marker bandis on an angiography image without contrast agent. In one embodiment,the user interface includes a longitudinal OCT image panel, across-sectional OCT image panel, one or more controls, and anangiography image panel. In one embodiment, the user interface includesa register control or button that causes the computing devices toexecute one or more software modules configured to co-register the OCTimage data and the angiography image data. In one embodiment, the timestamps are used to give a first-order match between angiography framesand their corresponding OCT frames, such that for every OCT frame, theclosest angiography frame can be located, and vice versa. In addition,time-stamped events, such as pullback start and stop, are also recordedto assist the co-registration process.

In one embodiment, a cursor or other identifier on the angiography imagedenotes the location of the OCT catheter reference markers coincidingwith the OCT pullback frame selected. In one embodiment, a cursor orother identifier can also denote the user-selected proximal and distalreference frames within which MLA has been calculated, and denote themean diameter of the blood vessel. Scrolling through the co-registeredOCT and angiography images can be controlled via the OCT L-mode or acursor on angiography frame as a remote controller or as part of theuser interface.

In one embodiment, a filter kernel such as a convolution matrix isimplemented as a matrix including rows and columns and elementsconfigured to perform image processing for performing intensifying,sharpening, pattern identification, detection, tracking and other imageprocessing tasks. The filter kernel can be used in various preprocessingand other processing stages to perform image processing on angiographyimage data or other image data.

In one embodiment, the invention relates to a processor-based method ofdisplaying an angiographic and an intravascular representation of ablood vessel. The method includes generating a set of OCT image data inresponse to distance measurements of a blood vessel using an opticalcoherence tomography system, the set comprising a plurality ofcross-sectional image at a plurality of positions along the bloodvessel; generating a set of angiography image data, the set comprising aplurality of two dimensional images at a plurality of positions alongthe blood vessel; and co-registering the angiography images and OCTimages based on one or more of a time stamp, a relationship between timestamps, matching of a feature in an OCT image with a feature in anangiography image, and determining a centerline for the blood vessel andusing the centerline to co-register the OCT images and angiographyimages.

In one aspect, the invention relates to a processor-based method ofdisplaying an angiographic and an intravascular representation of ablood vessel. The method includes generating a set of optical coherencetomography image data in response to distance measurements of the bloodvessel obtained during a pullback of a probe through the blood vesselusing an optical coherence tomography system, the set of OCT image datacomprising a plurality of cross-sectional image at a plurality ofpositions along the blood vessel; generating a set of angiography imagedata using an angiography system during the pullback of the probethrough the blood vessel using an optical coherence tomography system,the set of angiography image data comprising a plurality oftwo-dimensional images obtained at different points in time during thepullback; displaying a first panel comprising a first longitudinal viewof the blood vessel generated using the OCT image data; and displaying asecond panel comprising a frame of the angiography image dataidentifying the blood vessel using one or more points in the frame and avessel centerline passing through the one or more points.

In one embodiment, the method further includes co-registering the OCTimage data and the angiography data using vessel centerlines to create acontinuous registration of a tracked marker, wherein the tracked markeris disposed on an OCT data collection probe. In one embodiment, themethod further includes co-registering the OCT image data and theangiography data such that selecting a point along the vessel centerlinethrough a user interface changes a frame identifier in the firstlongitudinal view. In one embodiment, the method further includes usingpullback speed or pullback length to perform an iterative search toreject candidates for the tracked marker based on the possible locationsfor such markers based upon the pullback length and/or pullback speed.

In one embodiment, the vessel centerline is generated using a shortestpath technique and a plurality of processing steps from a Dijkstraalgorithm. In one embodiment, the method further includes the step ofremoving a guide catheter image from one or more frames of angiographydata using superposition of an intensity profile. In one embodiment, thevessel centerline is generated using path information generated from oneor more angiography frames substantially in the absence of contrastsolution. In one embodiment, the method 1 further includes generating aconfidence score for each detection and co-registration betweenangiography data and optical coherence tomography data.

In one aspect, the invention relates to a method of detecting anintravascular probe marker comprising obtaining a first frame ofangiography image data that is substantially free of contrast agentimage data and includes the intravascular probe marker; obtaining asecond frame of angiography image data that comprises contrast agentimage data in the vicinity of the intrasvascular probe marker; anddetecting the intravascular probe marker in the first frame and thesecond frame.

In one embodiment, the method further includes the steps of applying animage processing transform to the second frame to remove or modify afeature in the second frame and increasing an intensity of a pluralityof pixels, the plurality of pixels comprising a guidewire image in thesecond frame. In one embodiment, the method further includes the step ofgenerating an average intensity value for a plurality of images andsubtracting the average intensity from the first or second frame. In oneembodiment, the method includes applying a bottom hat operator to thesecond frame and applying a morphological close operation.

In one embodiment, detecting the intrasvascular probe marker comprisesfiltering candidate markers comprising pixels in the first frame and thesecond frame by applying a multiscale Laplacian of Gaussian operator onthe first frame and the second frame and performing a non-maximasuppression process to identify blobs having a relative maximum in aneighborhood of pixels.

In one embodiment, the method further includes the step of generating aguidewire-based potential function by applying a Euclidian distancetransform on a binary image. The method can also include applying anexponent to a negative fractional power times the distance transform tocompute the potential function. In one embodiment, the method furtherincludes determining a plurality of geodesic distances based on theguidewire-based potential using a fast marching method.

In one embodiment, the method further includes removing a shadow fromthe first frame and the second frame, increasing a contrast level of aguidewire on one of the first frame or second frame, and performing amorphological image reconstruction for each marker candidate. In oneembodiment, the method includes processing the plurality of pullbackframes using a Hessian-based vessleness filter; and tracking theintravascular probe marker from one of the first frame or the secondframe through the plurality of pullback frames to all the pullbackframes using template matching. In one embodiment, the method furtherincludes tracking the intravascular probe marker through a plurality offrames obtained during the pullback using a Viterbi dynamic programmingmethod.

In one aspect, the invention relates to a processor-based method ofco-registering angiographic image data and intravascular image dataobtained during a pullback through a blood vessel. The method includesstoring a plurality of frames of optical coherence tomography data inmemory; storing a plurality of frames of angiography image data inmemory; processing the plurality of frames of angiography image datasuch that one or more shadows are substantially reduced; detecting acatheter in the plurality of frames of angiography image data; removingthe detected catheter in the plurality of frames of angiography imagedata; generating a vessel centerline for the plurality of frames ofangiography image data; detecting a probe marker in the plurality offrames of angiography image data; tracking a position of the probemarker along one or more vessel centerlines; and co-registering theplurality of frames of angiography image data and the plurality offrames of optical coherence tomography data using the tracked position.

In one embodiment, the method includes generating a score indicative ofa level of confidence in co-registration between a frame of angiographyimage data and a frame of the optical coherence tomography data. In oneembodiment, the method includes removing the detected catheter isperformed using superposition of an intensity profile generated based ona sampling of regions of the detected catheter.

In one embodiment, the step of co-registering the plurality of frames ofangiography image data and the plurality of frames of optical coherencetomography data comprises generating a co-registration table, using acomputing device, the co-registration table comprising angiography imageframes, a plurality of per frame OCT time stamps, a plurality of perframe angiography time stamps, and optical coherence tomography imageframes. In one embodiment, the method further includes displaying astent representation in an OCT image and an angiography image in a userinterface using the co-registration table and a computing device.

In one embodiment, the method further includes identifying a side branchin one or more OCT images or angiography images using theco-registration table and a user interface configured to display theside branch. In one embodiment, the method further includes to set thespacing of the frames of OCT data based on the co-registration table toadjust for pullback speed changes and to display a longitudinal view ina user interface based on the spacing.

BRIEF DESCRIPTION OF THE DRAWINGS

The figures are not necessarily to scale, emphasis instead generallybeing placed upon illustrative principles. The figures are to beconsidered illustrative in all aspects and are not intended to limit theinvention, the scope of which is defined only by the claims.

FIG. 1 shows a schematic diagram of an angiography and intravascularimaging and data collection system in accordance with an illustrativeembodiment of the invention.

FIG. 2A shows a schematic diagram of a region of interest for a subjectand features of a catheter-based data collection probe in accordancewith an illustrative embodiment of the invention.

FIG. 2B shows a schematic diagram of a catheter-based data collectionprobe including a marker in accordance with an illustrative embodimentof the invention.

FIG. 3A shows an image of a graphic user interface suitable forcontrolling or reviewing data and images generated by the system of FIG.1 and/or the methods and software modules described herein in accordancewith an illustrative embodiment of the invention.

FIG. 3B shows an image of another graphic user interface suitable forcontrolling or reviewing data and images generated by the system of FIG.1 and/or the methods and software modules described herein in accordancewith an illustrative embodiment of the invention.

FIGS. 4A and 4B are schematic diagrams showing processing stages orprocessing steps suitable for processing and using image data inaccordance with an illustrative embodiment of the invention.

FIG. 5A shows a flow chart relating to some exemplary preprocessingsteps or stages in accordance with an illustrative embodiment of theinvention.

FIG. 5B shows a flow chart relating to some exemplary vessel centerlinegeneration steps or stages in accordance with an illustrative embodimentof the invention.

FIG. 5C shows a flow chart relating to some exemplary marker detectionand co-registration steps or stages in accordance with an illustrativeembodiment of the invention.

FIG. 6A is an exemplary bottom hat filter configured to enhance blobs orother pixel regions in an angiography image that are likely to be amarker from a probe in accordance with an illustrative embodiment of theinvention.

FIG. 6B is an exemplary blob corresponding to a subset of pixels from anangiography region that has been enhanced by the application of thefilter from FIG. 6A in accordance with an illustrative embodiment of theinvention.

FIG. 6C is an original angiography image without contrast agent prior towire detection in accordance with an illustrative embodiment of theinvention.

FIG. 6D is an exemplary angiography image showing the results ofguidewire detection on a frame without contrast agent in accordance withan illustrative embodiment of the invention.

FIG. 6E is an exemplary angiography image showing the results ofguidewire enhancement after the application of a bottom hat operator inaccordance with an illustrative embodiment of the invention.

FIG. 6F is an exemplary angiography image showing the results ofguidewire enhancement after the application of a Hessian operator havinga scale value of one in accordance with an illustrative embodiment ofthe invention.

FIG. 6G is an exemplary potential generated based on the guidewiresuitable for use with a fast marching method (FMM) process in accordancewith an illustrative embodiment of the invention.

FIG. 6H is a distance map generated using an FMM process in accordancewith an illustrative embodiment of the invention.

FIG. 6I is an original angiography image with contrast agent prior tocatheter and shadow removal in accordance with an illustrativeembodiment of the invention.

FIG. 6J is an exemplary angiography image showing the results ofcatheter and shadow removal in accordance with an illustrativeembodiment of the invention.

FIG. 6K is an original angiography image with contrast agent prior toshadow removal in accordance with an illustrative embodiment of theinvention.

FIG. 6L is an exemplary angiography image showing the results ofcatheter and shadow removal in accordance with an illustrativeembodiment of the invention.

FIG. 6M is an original angiography image with contrast agent prior toguide wire detection in accordance with an illustrative embodiment ofthe invention.

FIG. 6N is an exemplary angiography image showing the results ofguidewire detection with respect to the original image of FIG. 6V inaccordance with an illustrative embodiment of the invention.

FIGS. 7A-7F show the application of different software-based imageprocessing steps to generate a graph based upon a frame of angiographyimage data in accordance with an illustrative embodiment of theinvention.

FIGS. 8A-8C show various best paths found through the graph generated inFIG. 7F based on a graph searching algorithm in accordance with anillustrative embodiment of the invention.

FIGS. 9A-9E show various image processing stages relating to catheterdetection and removal in accordance with an illustrative embodiment ofthe invention.

FIGS. 10A-10B show an exemplary model of a catheter and the effect ofcontrast solution it its intensity profile in accordance with anillustrative embodiment of the invention.

FIGS. 11A-11B show features of using a superposition based catheterremoval method in accordance with an illustrative embodiment of theinvention.

FIG. 12 shows a schematic diagram of various software and hardwarecomponents suitable for processing intravascular and angiographic imagedata in accordance with an illustrative embodiment of the invention.

FIGS. 13A and 13B show a schematic diagram of exemplary OCT andangiography data tables for a pullback in accordance with anillustrative embodiment of the invention.

FIG. 14 shows a schematic diagram of an exemplary angiography data tablefor a pullback in accordance with an illustrative embodiment of theinvention.

FIG. 15 shows a schematic diagram of an exemplary co-registration tablein accordance with an illustrative embodiment of the invention.

DETAILED DESCRIPTION

The following description refers to the accompanying drawings thatillustrate certain embodiments of the present invention. Otherembodiments are possible and modifications may be made to theembodiments without departing from the spirit and scope of theinvention. Therefore, the following detailed description is not meant tolimit the present invention; rather, the scope of the present inventionis defined by the claims.

As described above, there are challenges relating to vascular andperipheral vascular diagnostic systems such as challenges relating toimplementing co-registration for multiple imaging technologies such asangiography, OCT, and IVUS. In part, the invention relates to varioussystems, components thereof, and methods for use in a catheter lab orother facility to collect data from a subject and help improve upon oneor more of these limitations. The data collected is typically related tothe patient's cardiovascular or peripheral vascular system and caninclude image data, pressure data, heart rate, and other types of dataas described herein.

In addition, in one embodiment image data is collected using opticalcoherence tomography probes and other related OCT components. In oneembodiment image data is collected using IVUS probes and other relatedIVUS components. In addition, in one embodiment pressure data iscollected using FFR probes and other related FFR components. Inaddition, in one embodiment EKG, heart rate, and other subject data iscollected using electrodes and other related components.

In addition, some embodiments of the invention are suitable for handlingmultiple imaging modalities. Thus, in part, the invention relates to amultimodal diagnostic system and components thereof configured toco-register one or more of the following OCT, IVUS, FFR, andangiography. OCT data and image processing results can be used toimprove the processing of frames of angiography images by providinginput into angiography specific software modules.

IVUS imaging features can also be incorporated into the data collectionprobe used in conjunction with collecting the angiography data in oneembodiment. Further, FFR pressure measurements can also be performedusing suitable pressure transducers and probes. In one embodiment, theFFR data collecting probes or transducers can include a wirelesstransmitter and employ a wireless receiver to receive and communicateFFR data to the server. Comparison and co-registration of OCT and/orIVUS images with angiographic images are achieved by interfacing thesystem with an angiography device or a hospital data network wherein theangiographic data is stored.

In one embodiment, a user such as a clinician interacts with aworkstation or server having an associated user interface for displayingimages of a subject's blood vessels from a top down, longitudinalcross-section, or a cross-section substantially parallel to thelongitudinal axis of the vessel. The co-registration process can includevarious steps and image processing and feature detection softwaremodules. In one embodiment, a user or a system activates intravascularimaging while acquiring angiographic images. The blood vessel beingimaged intravascularly and the imaging catheter can be displayed as partof a graphic user interface. The boundary of the lumen of the vessel canbe identified in each intravascular and angiography image and related toeach other to maintain the same vessel segment on different views.

Since the imaging catheter is introduced by a guidewire, the guidewirecan be used as an anchor path and to provide directional informationsuch as what endpoint is distal and what endpoint is proximal in therelevant imaging segment. In one embodiment, a guide catheter slidesalong the guidewire to position a probe tip having one or more imagingdevices in the blood vessel. In one embodiment, the angiographic imagedata is processed such that the guide catheter is removed from the imageafter it has been identified.

In one embodiment, one or more software modules are used to generate andtrack a vessel centerline for a given frame of angiography data. In oneembodiment, a vessel centerline also referred to herein as a centerlineis a model or simulation that is generated based on an iterativelyevaluation of each candidate subset of a frame of angiographic data formarker bands associated with the optical or acoustic sensor or otherimaging or data collecting sensor introduced during the angiographicdata collection. In one embodiment, a dynamic program software modulesuch as a software module implementing one or more steps of the Viterbialgorithm can be used to track the marker bands. In one embodiment, theViterbi algorithm is used for radiopaque marker tracking. The creationand tracking of the centerlines are typically handled by otheralgorithms or combinations thereof. In one embodiment, the vesselcenterlines are generated by a combination of algorithms or processesfor finding the shortest path between two far points such as a fastmarching algorithm on the Hessian image and a modified Dijkstraalgorithm.

FIG. 1 shows a system 5 which includes various data collectionsubsystems suitable for collecting data or detecting a feature of orsensing a condition of or otherwise diagnosing a subject 10. In oneembodiment, the subject is disposed upon a suitable support 12 such astable bed to chair or other suitable support. Typically, the subject 10is the human or another animal having a particular region of interest25.

In part, embodiments of the invention relate to co-registration ofintravascular images or data acquired by an imaging catheter whichtraverses a blood vessel, and external angiographic images of thatvessel taken at the time of the catheter's traversal. A magnified,although also a generalized schematic view, of the region of interest isshown in FIG. 2A.

In a typical OCT data acquisition procedure, a catheter is inserted overa guidewire to steer the probe to the distal end of a target bloodvessel. The probe 30 can include one or more markers. In one embodiment,the marker disposed on the probe 30 is a radiopaque marker band. Thetorque wire 110, which partially surrounds optical fiber 33, is alsoshown in FIG. 2A. The probe 30 is disposed in the lumen 50 of the bloodvessel. A guidewire 115 is also shown in the lumen 50. The guidewire 115is used to position the probe tip and the torque wire which are disposedin a catheter to the lumen. Light A, from the probe tip is shown beingdirected to the wall of the blood vessel having lumen 50.

Additional details relating to an exemplary intravascular datacollection probe is shown in FIG. 2B. As shown in FIG. 2B, anintravascular data collection probe 120 such as an OCT, IVUS, FRR, orother data collection probe, includes an optical fiber 33 configured todirect light as shown by the dotted line as part of a probe tip. Asheath such as a polymer sheath 125 surrounds the probe tip whichincludes a beam directing element such as lens or a reflector. Light A,is shown exiting the beam director along the dotted line. The opticalfiber 33 is disposed in a torque wire 110 which is also disposed withinthe sheath 120. The optical fiber 33 is coupled to PIU 35 as shown.

As shown in FIG. 2B, a marker or marker band 130 such as a radiopaquemarker is part of the data collection probe 120. The markers aredetectable by angiography systems and can be tracked as they move acrossframes of angiography data. As shown, the distance from the right edgeof the torque wire 127 to the beam directing element such as lens or areflector is L1.

Additionally, the distance from the right edge of the torque wire 127 tothe right edge of the marker 130 is L2. The thickness of the marker 130is L3. The distance from the distal edge of the marker 130 (shown asleft side of marker) to the torque wire 127 is L3+L2. In one embodiment,L1 ranges from about 0.3 mm to about 0.9 mm. In one embodiment, L2ranges from about 0.6 mm to about 1.4 mm. In one embodiment, L3 rangesfrom about 0.5 mm to about 1.5 mm.

In one embodiment, a data collection probe such as an OCT probe caninclude three radiopaque marker bands. The distal marker located at thedistal end of the probe remains stationary throughout the acquisition.The middle marker is located at the imaging core, which resides 27 mmfrom the distal marker before pullback. The proximal marker is located50 mm from the imaging core and this distance remains fixed during thepullback.

During the pullback, a processor-based system, such as system 22 in FIG.1, records live angiograms, and displays blood vessels with a contrastagent and the marker or the probe. Typically, the markers are visiblemost of the time. Optionally, some frames are recorded without anycontrast agent, such as shown in FIG. 6C, such that the guidewire andmarkers are clearly visible. This provides a good indication of thepullback track through the vessel.

FIG. 3A shows an exemplary graphic user interface configured to displaymultiple panels. The graphic user interface can be implemented using acomputing device such as the server 50 or workstation 87 or anothersuitable computing device. The upper right panel shows frame angiographyimage data. As shown in the image, a section of a blood vessel disposedbetween an upper point or cursor 3 and a lower point or cursor 4 wasimaged using an intravascular imaging technology as part of a pullback.Specifically, the angiographic data was obtained while an OCT pullbackwas performed.

An exemplary cross-section of the artery is shown in the upper leftpanel. In the upper left OCT image side branch is shown to the right ofthe cross-section of the data collection probe. Lower panel, whichsubstantially spans the user interface, includes the longitudinal imageof the blood vessel disposed between the distal end point and theproximal end point shown in the angiography image shown by points orcursors 3, 4. The magnifying glass icon can be used to zoom in or out oneither the OCT or angiography image. The pencil icon can be used to makemeasurements on either the OCT or angiography image. The angiographyframes of data can be played as video in the upper right panel by usingthe play, review, or forward video user interface controls.

In the upper left OCT image, the angled axis shows the cut plane used todisplay the longitudinal mode in the lower panel. The longitudinal modeis generated by combining a plurality cross-sectional view such as thatshown in the upper left quadrant interface. In the L mode the triangle4′ is configured to show a bookmarked location of a frame of interest.Longitudinal view or L mode can be advanced or reviewed or shown in ananimated manner using the review, play, and forward L mode userinterface but the vertical line shown in the L mode corresponds to thecross-sectional slice of the blood vessel shown in the cross-sectionalOCT image above. By selecting the play and review buttons in the L modethe corresponding vertical line advances or retreats as differentcross-sections are shown in the upper OCT image as the vertical linemoves in the L mode in the lower panel.

In one embodiment, the computing device used to display and execute theuser interfaces of FIGS. 3A and 3B includes memory storage whichincludes image data such as cross-sectional views of a blood vessel. Thecomputing device can include machine readable medium or other memorythat includes one or more software modules for displaying a graphicaluser interface such as interface 142. The interface can include aplurality of panels, menus or other displayable regions. These panels orregions can be displayed on one or more monitors such as display 82. Thecomputing device can exchange data such as image data with the monitor23 using a network which can include one or more wired, optical,wireless or other data exchange connections.

A controller or input device 127 can be in wired, optical, or otherwisein communication with the other devices or systems shown over thenetwork 120. The controller can be used to send command signals to thecomputing system 100 which is running the interface 142. The interface142 can display data from the system 5 of FIG. 1, system 300 of FIG. 14,or other sources of data, systems or software modules described herein.The interface 142 can include one or more menus and other sections thatchange in response to control signals from controller 127. Thecontroller 127 can include a processor or suitable programmable ASIC.The control signals can be sent over the network 120 or via anotherconnection.

The computing device 100 may include a server computer, a client usercomputer, a personal computer (PC), a laptop computer, a tablet PC, adesktop computer, a control system, a microprocessor or any computingdevice capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that computing device.Further, while a single computing device is illustrated, the term“computing device” shall also be taken to include any collection ofcomputing devices that individually or jointly execute a set (ormultiple sets) of instructions to perform any one or more of thesoftware features or methods such as interface 142.

FIG. 3B shows a representation of a graphic user interface 142. Theinterface 142 includes a plurality of panels. As shown, there are fourmain panels 150, 155, 160, and 165 in one embodiment. These include anauxiliary display panel 150 which shows angiography data in thisembodiment, a cross-sectional view or B mode display panel 155, a lumenprofile panel 160, and an L mode display panel 165. In one embodiment,the interface also includes multiple toolbars B1, B2, and B3. In panel150, three markers are shown as crosses superimposed over theangiography image. The top marker corresponds to a proximal referenceframe shown in panel 160. The middle marker corresponds to a minimumlumen area frame shown in panel 160 or an active OCT frame shown inpanel 155. The bottom marker corresponds to a distal reference frameshown in panel 160. The angiography frames and OCT frames of image datathat can be displayed using interfaces in FIGS. 3A and 3B can beprocessed and co-registered as outlined herein. In one embodiment, thecommuting device accesses a co-registration table to display theco-registered frames.

FIG. 3B shows a minimum lumen area plot as part of the lumen profile forthe blood vessel imaged during a pullback of the OCT probe in panel 160.The D and P arrows show proximal and distal directions along the imagedblood vessel. The cut plane shown as a line having sections L1 and L2 isshown in the cross-sectional view of panel 155 and also shown bysections L1 and L2 in the L-mode panel 165. An information bar B1, ameasurement bar B2, and a menu bar B3 are shown.

As shown, the distance of a blood vessel such as an artery can bemeasured relative to two endpoints as shown by the exemplary measurementdistances of 119.88 mm. In addition, the mean diameter can be shown ateach end of the selected reference frames for measuring the vessel suchas by the mean diameter values of 39.2 mm and 44.2 mm at the distal andproximal reference frames respectively. As shown, the MLA is about 22mm². At the MLA frame, the vessel mean diameter is about 2.11 mm and thepercent diameter stenosis is 25.4% relative to the average diameters ofthe proximal and distal reference frames.

All three images shown in the user interface of FIGS. 3A and 3B areco-registered such that movement along the line between the ends of theblood vessel in the angiographic image can be shown by a moving pointthat synchronizes with the frames in the OCT images. Accordingly as onemoves along the blood vessel segment, movement along the centerlineshown in the angiographic image is also shown by a moving frameidentifier in the cross-sectional OCT image or the L mode OCT image orboth.

Initially, the proximal marker band may reside near the ostium of thecoronary branch, thus it is occluded by a cloud of contrast agent duringthe pullback. The catheter is pulled back at constant speed through thevessel. Due to different foreshortening of blood vessel segments alongthe pullback, the marker does not move at constant speed in theangiography image plane (2D). Furthermore, due to the cardiac motion,the marker exhibits a distinctive “sawing” motion relative to theanatomy of the vessel. In some of the angiography frames, the markerbands appear blurred/faint due to fast pullback motion combined withfast cardiac motion. The contrast of the marker in the localneighborhood might be low. Other features, such as foreshortenedbifurcations, background structures and the like, may be mistaken forany of the marker bands.

The data collection system 5 includes a noninvasive imaging system suchas a nuclear magnetic resonance, x-ray, computer aided tomography, orother suitable noninvasive imaging technology. As shown as anon-limiting example of such a noninvasive imaging system, anangiography system 20 such as suitable for generating cines is shown.The angiography system 20 can include a fluoroscopy system. Angiographysystem 20 is configured to noninvasively image the subject 10 such thatframes of angiography data, typically in the form of frames of imagedata, are generated while a pullback procedure is performed using aprobe 30 such that a blood vessel in region 25 of subject 10 is imagedusing angiography in one or more imaging technologies such as OCT orIVUS, for example.

The angiography system 20 is in communication with an angiography datastorage and image management system 22, which can be implemented as aworkstation or server in one embodiment. In one embodiment, the dataprocessing relating to the collected angiography signal is performeddirectly on the detector of the angiography system 20. The images fromsystem 20 are stored and managed by the angiography data storage andimage management 22. In one embodiment system server 50 or workstation87 handle the functions of system 22. In one embodiment, the entiresystem 20 generates electromagnetic radiation, such as x-rays. Thesystem 20 also receives such radiation after passing through the subject10. In turn, the data processing system 22 uses the signals from theangiography system 20 to image one or more regions of the subject 10including region 25.

As shown in this particular example, the region of interest 25 is asubset of the vascular or peripherally vascular system such as aparticular blood vessel. This can be imaged using OCT. A catheter-baseddata collection probe 30 is introduced into the subject 10 and isdisposed in the lumen of the particular blood vessel, such as forexample, a coronary artery. The probe 30 can be a variety of types ofdata collection probes such as for example an OCT probe, an FFR probe,an IVUS probe, a probe combining features of two or more of theforegoing, and other probes suitable for imaging within a blood vessel.The probe 30 typically includes a probe tip, one or more radiopaquemarkers, an optical fiber, and a torque wire. Additionally, the probetip includes one or more data collecting subsystems such as an opticalbeam director, an acoustic beam director, a pressure detector sensor,other transducers or detectors, and combinations of the foregoing.

For a probe that includes an optical beam director, the optical fiber 33is in optical communication with the probe with the beam director. Thetorque wire defines a bore in which an optical fiber is disposed. InFIG. 1, the optical fiber 33 is shown without a torque wire surroundingit. In addition, the probe 30 also includes the sheath such as a polymersheath (not shown) which forms part of a catheter. The optical fiber 33,which in the context of an OCT system is a portion of the sample arm ofan interferometer, is optically coupled to a patient interface unit(PIU) 35 as shown.

The patient interface unit 35 includes a probe connector suitable toreceive an end of the probe 30 and be optically coupled thereto.Typically, the data collection probes 30 are disposable. The PIU 35includes suitable joints and elements based on the type of datacollection probe being used. For example a combination OCT and IVUS datacollection probe requires an OCT and IVUS PIU. The PIU 35 typically alsoincludes a motor suitable for pulling back the torque wire, sheath, andoptical fiber 33 disposed therein as part of the pullback procedure. Inaddition to being pulled back, the probe tip is also typically rotatedby the PIU 35. In this way, a blood vessel of the subject 10 can beimaged longitudinally or via cross-sections. The probe 30 can also beused to measure a particular parameter such as an FFR or other pressuremeasurement.

In turn, the PIU 35 is connected to one or more intravascular datacollection systems 40. The intravascular data collection system 40 canbe an OCT system, an IVUS system, another imaging system, andcombinations of the foregoing. For example, the system 40 in the contextof probe 30 being an OCT probe can include the sample arm of aninterferometer, the reference arm of an interferometer, photodiodes, acontrol system, and patient interface unit. Similarly, as anotherexample, in the context of an IVUS system, the intravascular datacollection system 40 can include ultrasound signal generating andprocessing circuitry, noise filters, rotatable joint, motors, andinterface units. In one embodiment, the data collection system 40 andthe angiography system 20 have a shared clock or other timing signalsconfigured to synchronize angiography video frame time stamps and OCTimage frame time stamps.

In addition to the invasive and noninvasive image data collectionsystems and devices of FIG. 1, various other types of data can becollected with regard to region 25 of the subject and other parametersof interest of the subject. For example, the data collection probe 30can include one or more pressure sensors such as for example a pressurewire. A pressure wire can be used without the additions of OCT orultrasound components. Pressure readings can be obtained along thesegments of a blood vessel in region 25 of the subject 10.

Such readings can be relayed either by a wired connection or via awireless connection. As shown in a fractional flow reserve datacollection system 45, a wireless transceiver 47 is configured to receivepressure readings from the probe 30 and transmit them to a system togenerate FFR measurements or more locations along the measured bloodvessel. One or more displays 82 can also be used to show an angiographyframe of data, an OCT frame, user interfaces for OCT and angiographydata and other controls and features of interest.

The intravascular image data such as the frames of intravascular datagenerated using the data collection probe 30 can be routed to the datacollection processing system 40 coupled to the probe via PIU 35. Thenoninvasive image data generated using angiography system 22 can betransmitted to, stored in, and processed by one or more servers orworkstations such as the co-registration server 50 workstation 87. Avideo frame grabber device 55 such as a computer board configured tocapture the angiography image data from system 22 can be used in variousembodiments.

In one embodiment, the server 50 includes one or more co-registrationsoftware modules 60 that are stored in memory 70 and are executed byprocessor 80. The server 50 can include other typical components for aprocessor-based computing server. Or more databases such as database 90can be configured to receive image data generated, parameters of thesubject, and other information generated, received by or transferred tothe database 90 by one or more of the systems devices or componentsshown in FIG. 1. Although database 90 is shown connected to server 50while being stored in memory at workstation 87, this is but oneexemplary configuration. For example, the software modules 60 can berunning on a processor at workstation 87 and the database 90 can belocated in the memory of server 50. The device or system use to runvarious software modules are provided as examples. In variouscombinations the hardware and software described herein can be used toobtain frames of image data, process such image data, and register suchimage data.

As otherwise noted herein, the software modules 60 can include softwaresuch as preprocessing software, transforms, matrices, and othersoftware-based components that are used to process image data or respondto patient triggers to facilitate co-registration of different types ofimage data by other software-based components 60 or to otherwise performsuch co-registration.

The database 90 can be configured to receive and store angiography imagedata 92 such as image data generated by angiography system 20 andobtained by the frame grabber 55 server 50. The database 90 can beconfigured to receive and store OCT image data 95 such as image datagenerated by OCT system 40 and obtained by the frame grabber 55 server50. The database 90 can be configured to receive and store anangiography table such as that shown in FIG. 14 and a co-registrationtable such as that shown in FIG. 15.

In addition, the subject 10 can be electrically coupled via one or moreelectrodes to one more monitors such as, for example, monitor 49.Monitor 49 can include without limitation an electrocardiogram monitorconfigured to generate data relating to cardiac function and showingvarious states of the subject such as systole and diastole. Knowing thecardiac phase can be used to assist the tracking of vessel centerlines,as the geometry of the heart, including the coronary arteries, isapproximately the same at a certain cardiac phase, even over differentcardiac cycles.

Hence, if the angiography data spans a few cardiac cycles, a first-ordermatching of vessel centerline at the same cardiac phase may assist intracking the centerlines throughout the pullback. In addition, as mostof the motion of the heart occurs during the systole, vessel motion isexpected to be higher around the systole, and damp towards the diastole.This provides data to one or more software modules as an indication ofthe amount of motion expected between consecutive angiography frames.Knowledge of the expected motion can be used by one or more softwaremodules to improve the tracking quality and vessel centerline quality byallowing adaptive constraints based on the expected motion.

The use of arrow heads showing directionality in a given figure or thelack thereof are not intended to limit or require a direction in whichinformation can flow. For a given connector, such as the arrows andlines shown connecting the elements shown in FIG. 1, for example,information can flow in one or more directions or in only one directionas suitable for a given embodiment. The connections can include varioussuitable data transmitting connections such as optical, wire, power,wireless, or electrical connections.

Furthermore, although the FFR data collection system 45 is shown ashaving a wireless system 47 suitable for sending and receivinginformation wirelessly, the other systems and components shown in FIG. 1also include wireless systems such as system 47 and can send and receiveinformation wirelessly in one embodiment.

One or more software modules can be used to process frames ofangiography data received from an angiography system such as system 22shown in FIG. 1. Various software modules which can include withoutlimitation software, a component thereof, or one or more steps of asoftware-based or processor executed method can be used in a givenembodiment of the invention.

Examples of such software modules can include without limitation a videoprocessing software module, a preprocessing software module, an imagefile size reduction software module, a catheter removal software module,a shadow removal software module, a vessel enhancement software module,a blob enhancement software module, a Laplacian of Gaussian filter ortransform software module, a guidewire detection software module, ananatomic feature detection software module, stationary marker detectionsoftware module, a background subtraction module, a Frangi vesselnesssoftware module, an image intensity sampling module, a moving markersoftware detection module, iterative centerline testing software module,a background subtraction software module, a morphological closeoperation software module, a feature tracking software module, acatheter detection software module, a bottom hat filter software module,a path detection software module, a Dijkstra software module, a Viterbisoftware module, fast marching method based software modules, a vesselcenterline generation software module, a vessel centerline trackingmodule software module, a Hessian software module, an intensity samplingsoftware module, a superposition of image intensity software module andother suitable software modules as described herein. The software module60 shown in FIG. 1 can include one or more of the foregoing softwaremodules and other software modules described herein.

Image Data Processing Features and Exemplary Embodiments

As shown in FIGS. 4A and 4B, various processing stages, steps orsoftware modules are generalized to provide a high level summary of theprocess of co-registering angiography image data and image data obtainedusing an intravascular imaging technology such as OCT, IVUS, or others.In one embodiment, frames of angiography data are captured on an OCT orIVUS server or workstation using a frame grabber or other data capturedevice. Capturing images from both imaging modalities in real timeensures accurate time stamping of the two sources with respect to oneanother. DICOM angiography data acquisition time cannot be inherentlycalibrated to match the timing of the OCT data. For example, a videosoftware module can be controlled via a user interface to presentangiography video to a frame grabber which can in turn obtain and storeindividual frames of angiography data with a time stamp. In oneembodiment, the OCT data and the angiography data are date stamped bytwo respective processes that run in parallel on the same computer andhence share the same time base.

Once the angiography data frames have been cached or otherwise stored,each of the stored frames can be modified during a preprocessing stage.Various matrices such as convolution matrices, Hessians, and others canbe applied on a per pixel basis to change the intensity, remove, orotherwise modify a given angiography image frame. As discussed herein,the preprocessing stage effectively enhances or modifies or removesfeatures of the angiography images to increase the accuracy, processingspeed, success rate, and other properties of subsequent processingstages.

As shown in FIG. 4A, various software-based processing stages 140 areshown. Initially, one or more frames of angiography images are processedduring a preprocessing stage 140 a prior to various detection andtracking stages in support of co-registering such frames with otherimage data obtained with another imaging technology such as OCT, IVUS,others, and combinations thereof. The next stage is a vessel centerlinedetermination or calculation stage 140 b. As shown in the user interfaceof FIG. 3, a vessel centerline is generated by one or more softwaremodules and superimposed or otherwise displayed relative to theangiography image.

In one embodiment, the centerline represents a trajectory of the probesuch as the data collection probe 30 of FIG. 1 through the blood vesselbeing imaged during the pullback. In one embodiment, the centerline isalso referred to as a trace. Another stage is the detection of markerband in angiography frames 140 c. In one embodiment, the last stage is aco-registration stage. These stages and the other stages and methodsdescribed herein can be performed in different orders, interactively, inparallel or in series or combinations thereof. Additional steps andstages can also be added before or after or in between a given stage orstep. Additional examples of exemplary stages and steps reciting furtherdetails are shown in FIGS. 4B and 5A-5C.

As shown in FIG. 4B, various software-based processing stages orprocessing steps 145 are shown that include further detail relative tothose shown in FIG. 4A. Initially, preprocessing of angiography framesis performed 150 a. Detecting of guidewire on a frame without contrastagent is performed 150 c as shown in FIG. 6D. FIG. 6N is an exemplaryangiography image showing the results of guidewire detection. As shownin FIG. 6N, the distal part of the guidewire is detected.

Next, in one embodiment, generating vessel centerline on one frame isperformed 150 e. In one embodiment, a user input such as the selectionof a guidewire endpoint in the lumen being imaged via a user interfaceis stored as a user selected end point alternatively referred to as ahint point. Such a hint point can be used to generate the vesselcenterline on one frame such that a trace between the hint point and adistal point is generated for the relevant frames of angiography data.In one embodiment, such a relevant frame is obtained without contrastsolution being disposed in the blood vessel.

Still referring to FIG. 4B, tracking of vessel centerlines alongangiography frames is performed 150 f. In one embodiment, such trackingof vessel centerlines is performed with regard to all or substantiallyall of the angiography frames obtained during the pullback. Radio-opaquemarker tracking and/or marker detecting in angiography frames isperformed 150 h. In one embodiment, a Viterbi algorithm is used toperform marker tracking Co-registering OCT images and angiography imagesis performed 150 j. Generating a confidence score/figure of merit isperformed 150 l.

Generating a confidence score/figure of merit (FOM) is performed usingone or more software modules 1501. In one embodiment, the confidencescore or (FOM) is provided to a user by graphical representation on acomputer monitor, for example by providing a color-code on the X-ray orOCT image indicating regions of the OCT pullback that have high or lowconfidence of being co-registered. Regions of low confidence may, forexample, be indicated by a red strip or bar on the X-ray image near thevessel segment where low FOM values were obtained. The FOM/Scorereflects a confidence measure in the returned results. The score is inthe range of [0, 1] where 0 reflects the lowest confidence and 1reflects the highest. A FOM threshold value can be selected to define aboundary between high confidence and low confidence co-registrationresults. The threshold value can be chosen to give a desired sensitivityand specificity for identifying high-error locations by producing areceiver-operator curve (ROC). If low FOM values are obtained for alarge portion of the frames in a given pullback, such that the overallquality of the co-registration is questionable, no co-registrationresults may be displayed to the user.

The FOM determination is a scoring process that is based upon one ormore factors such as the quality of the detected blob (contrast orintensity of detected blob compared to that of immediate neighborhood,shape, size, etc.), the distance of the detected blob from its nominallyexpected position (based on pullback speed, frame rate calculations),the number of blob candidates that were found in the same vicinity (themore candidates, the lower the FOM), and intensity-based z-score, theoverall score of the Viterbi algorithm (how well the overall collectionof detected blobs represents a pullback) and other factors and measures.In one embodiment, a weighted average including one or more of theparameters recited herein can be used to generate a FOM or score.

The various steps and stages shown in FIG. 4A and FIG. 4B and asotherwise described herein can be performed automatically in whole or inpart in various embodiments. Additional details relating to somespecific examples of some of the steps and methods of FIG. 4A and FIG.4B are described herein, such as with respect to FIGS. 5A-5C. Forexample, FIG. 5A shows a flow chart relating to some exemplarypreprocessing steps or stages.

Exemplary Angiography Image Data Preprocessing Embodiments

In part, as shown in FIGS. 4A and 4B and otherwise described herein, inpart, the invention includes one or more preprocessing stages,preprocessing software modules, and related methods with regard to thecollected frames of angiography data. In one embodiment, imagepreprocessing is performed on a per frame basis with respect to theframes of angiography image data such as the data generated by system 20of FIG. 1. The preprocessing stage can include, without limitation,methods, stages, and software components, and other components suitableto perform vessel enhancement, catheter removal, shadow removal, heartshadow removal, blob enhancement such as by applying a multiscaleLaplacian of Gaussian, detection of anatomic features, skeletongeneration, angiography image size reduction, background subtraction,bottom hat filters, and others.

Various matrices such as Hessians and other types of filters and maskscan be applied to enhance the frames of angiography data prior to thembeing subjected to further processing to track markers, generatecenterlines, be co-registered with OCT, IVUS, or other images or data.One or more image processing stages can be used to preprocess frames ofangiography data received from an angiography system such as system 22or the server or workstation 50 and 87 shown in FIG. 1.

FIG. 5A shows a process flow 160 relating to some additional specificexemplary preprocessing steps or stages. As shown, angiography imagescan be processed at various stages in parallel. In one embodiment, LoGfiltering is performed at multiple scales 160 a. Each scale correspondsto size of an element in the image that will be acted upon by thefilter. A LoG multiscale based filter can be used, in one embodiment, toenhance blobs corresponding to the moving marker on the imaging probe.Different scales are used because of the different sizes of the markers.In one embodiment, to be sensitive to different sizes of the blobs, andless sensitive to noise, the LoG operator is computed at several scales.An example of a LoG filter is shown in FIG. 6A. An example of a blob as(a set of pixels from an angiography image) corresponding to a markerthat has been enhanced from applying the LoG of FIG. 6A as part of animaging processing software enhancement is shown in FIG. 6B. In oneembodiment, background subtraction to reduce the effect of staticfeatures based on an average of several frames of angiography images isperformed.

In addition, in one embodiment, a bottom hat filter or transform 160 ccan be applied to the angiography data to increase the visibility of theguidewire in the image. In one embodiment, the bottom hat filter isconfigured to erase features larger than the size of particularstructural element in a given angiography figure such as the diaphragm,skeletal features, etc. An example of a bottom hat filter or bottom hatoperator applied to an angiography image is shown in FIG. 6E. In oneembodiment, multiple image averaging is used for background subtraction.In addition, in one embodiment Hessian filtering at a scale, such asscale 1, is performed 160 e following the bottom hat filter ortransform. Such a Hessian filter at scale 1 is performed in order toenhance the wire, while smoothing the noisy image after the applicationof the bottom hat operator. An example of a Hessian filter at scale 1applied to an image is shown in FIG. 6F.

In one embodiment, a morphologic close operation is performed on theimage data. The morphologic close operation is mainly used to fill inpossible gaps, sometimes obtained in the step of applying the bottom hattransform. The bottom hat transform is applied with a small filterkernel in order to enhance narrow features such as a guidewire.

Binary Image Map Features

For each angiography image, a set of preprocessing steps is applied tocreate a binary map which is used for determining where contrast agentis present. In one embodiment, a binary map refers to an image the samesize as the original angiography image, where a pixel is either black orwhite—black for a pixel with dye, white for pixel without dye or viceversa. The binary map may have areas of vessel pixels separated due tothe inherent imperfection of the binary map. A distance map can then becomputed based on the binary map. An exemplary distance map is shown inFIG. 6H, which was computed using an FMM algorithm.

A distance map is an image the same size, where the value of each pixelis determined according to its distance from the closest “black” pixelin the binary map. Clearly, the pixels where dye was determined to bepresent in the binary map (the “black” pixels—for which the distancefrom a dye area is 0) will remain black, the pixels immediatelysurrounding an area of black pixels (for whom the distance from a dyearea is 1) will have intensity lower by “1”. The next layer of pixels'intensity will be lower by “2”, etc. As shown in FIG. 6H, variousintensity values are mapped to pixels arranged along x and y axis forthe pixel locations. A scale coded by color or other indicia can be usedto map intensity values to each pixel location. In one embodiment, thescale is a color scale. Various exemplary intensity values on the scaleare shown in the figure. The central region has the lowest intensityvalues corresponding to B. The T intensity values increase relative tothe B values. The Y intensity values increase relative to the T valuesand the R values increase relative to the Y intensity values.

The resulting distance map is such that the areas of dye/contrast agentin the original binary map will look like ridges, with slopes going downto their sides. If two such ridges are close enough (small distance inthe binary map) they will appear as connected ridges in the distancemap. The dark central spot with the smallest value in the distance mapbelongs to the user hint point from where the front starts to propagate.Due to the configuration of the potential, it propagates along the wire.The distal end point of the trace has the highest value on the distancemap. One application of a distance map is to decide which separatesegments of dye/contrast agent can be connected since they are closeenough. In one embodiment, a distance map is a tool that is used todetermine the vessel skeleton from the binary map. The distance map canbe used for various purposes.

Exemplary Anatomic Feature Detection/A Priori Data GenerationEmbodiments

Further, in one embodiment, as part of the preprocessing of theangiography images, anatomic feature detection is performed. In oneembodiment, this can be performed to generate certain a prioriinformation relating to the path the imaging probe takes through theblood vessel. The generation of line segments such as through a skeletongeneration process can be used for feature detection. In one embodiment,a skeleton is a static object such as one or more line segments createdto help trace the blood vessels of a subject being imaged.

The use of a skeleton or line segment based approach to generate acandidate path through the blood vessel for the data collection probewhich can be used to inform centerline generation and marker trackingoffers several advantages to forgoing the use of such an approach. Forexample, the skeleton based approach can prevent or eliminate certaincenterline traces being generated that would otherwise pass through aside branch or the imaging probe catheter. Generating skeletons providesa method to determine an initial candidate for the geometry of the bloodvessel being imaged and side branches and other blood vessels as a mapor framework to facilitate centerline generation. By generatingskeletons, it is possible to extract points of interest such asbifurcation points and vessel segments, to stabilize tracking of markersand centerline traces and to verify tracking quality across frames ofangiography image data.

In one embodiment, the process of generating skeletons to detectanatomic features like side branches and vessel geometry is implementedduring preprocessing of the angiography images 160 d. Skeletons can beused for detecting anatomical features such as main bifurcation (170 l)and extrapolation point (170 m). In addition, skeletons can be used fordetecting and generating a smooth vessel centerline (170 f). Forexample, skeletons can be used with the Dijkstra algorithm. Theskeletons can be generated based on preprocessed Hessian images. A userselected point on an angiography image, such as the image of FIG. 7A,relating to a guidewire position can be used to reduce noise andfacilitate skeleton generation.

In FIG. 7D, a user selected end point and a computer determined endpoint are shown by the X's. A binary image generated from the Hessianimage can be used to generate skeletons in the angiography image asshown in FIG. 7B. Once generated, the skeletons can be eroded toeliminate small bifurcations. For example, small branches of theskeleton can be removed or subtracted from the image until only a maintrunk section remains. Thresholds relating to branch thickness and otherparameters can be used to direct skeleton erosion. The removal of smallbranches of the skeleton can be performed on a per pixel basis in oneembodiment until final skeleton results as shown in FIG. 7C.

In one embodiment, junctions are located on the skeleton by detectingbifurcations and other gaps as shown by the circled regions in FIG. 7D.These junctions are used to decompose the skeleton into branches asshown by branches 1-13 in FIG. 7E. In turn, each branch of the tree thatis too small to represent a vessel branch is eroded and can beeliminated. In one embodiment, all branches are eroded equally (by thesame number of pixels in length). As a result, the longer ones survivewhile the small ones are eliminated. The remaining skeleton branches canthen be transformed into to a connected graph as shown in FIG. 7F. Thedistance between graph nodes, i.e., the skeleton branches, such as nodes2 and 4 in FIG. 7F, is based on angle changes. For i=2 and j=4 for thenodes the following distance relationship can be used:d(t,f)=Δ(θ_(t,out),θ_(f,in))to obtain d(2,4) as shown in FIG. 7F. In one embodiment, a graphsearching method such as the Dijkstra shortest path algorithm ormodified versions thereof is applied to the graph to obtain bestcandidate paths for the blood vessel in the skeleton. This is actually amodified version of the Dijkstra algorithm. The chosen path is the pathbetween nodes in which the maximal angle change was the smallestregarding the other optional paths such as provided for by:path=min(max(dθ_(ƒ)|ƒεpath function))

FIGS. 8A-8C show the resulting best paths found relative to the skeletonof FIG. 7E based on the application of the Dijkstra shortest pathalgorithm to the graph generated in FIG. 7F. FIG. 8A shows a paththrough nodes 2, 4, 8, 7, 6, 3, and 1. FIG. 8B shows a path throughnodes 4, 6, 9, 3, and 1. FIG. 8C shows a path through nodes 2, 6, 8, 9,7, 5, 3, and 1. The use of the angles for distance measurements isuseful given the three-dimensional nature of how the nodes and branchesare arranged in a subject.

Exemplary Catheter Detection Embodiments

Further, in one embodiment, as part of the preprocessing of theangiography images, catheter detection is performed 160 f. The presenceof the catheter in the field of view may interfere with various stepsand processing stages of a co-registration method. An intersectionbetween the catheter and the vessel may be interpreted as a falsebifurcation, which can lead to unstable tracking. Tracking of markersand centerlines can be negatively affected by the presence of thecatheter delivering the intravascular imaging device. Another problemassociated with such a catheter is that the shortest path between twopoints along the vessel may be passed through the catheter, instead ofthe vessel. As a result, the catheter can lead to error and falsecenterline generation.

Therefore, it is desirable to be able to remove the catheter from eachframe of angiography data prior to pursuing subsequent processing anddetection such as in support of centerline generation. With respect to agiven input angiography image, such as shown in FIG. 9A, a vector fieldsuch as shown in FIGS. 9B-9D can be superimposed on the image based onthe detection of which sections of the image are moving and whichsections of the image exhibit a directional field as shown in FIG. 9Bwith the catheter spanning the middle portion of the figure and theblood vessel crossing it at an angle roughly in the middle of thefigure. FIG. 9C shows a vector field map of a vessel area while FIG. 9Dshows the substantially straight and vertically directed vectors in thecatheter area.

The vectors in the vector field illustrated in FIGS. 9C and 9D are theeigenvectors corresponding to the eigenvalues of the Hessian matrixcomputed by local second order analysis. In FIG. 9C, all the scales from1 to 5 were used in a Frangi Filter. An example of such a filter isdescribed in A. F. Frangi, W. J. Niessen, K. L. Vincken, M. A.Viergever, “Multiscale vessel enhancement filtering”, MICCAI'98, pp.130-137, and thus the turbulent influences outside the vessel. In FIG.9D, only scale sigma=4 was used and thus the isolated orientation on thecatheter, while in the outer regions, the eigenvectors have zeroweights. With regard to the sigma parameter, this parameter representsthe scale of the Gaussian used in the convolution computation. Sigma=4reflects the typical width in pixels for the catheter, as observed inthe angiography dataset.

In one embodiment, catheter detection is based on a primary assumptionof directionality of the catheter and on the fact that the catheteralways intersects the lower boundary of the image such as shown in FIG.9D. Though locally, the catheter and the vessel are generally notdistinguishable from each other given their tubular structure. In termsof the shape of the catheter, the catheter can be differentiated fromthe vessel globally because it crosses almost the entire image and has asubstantially straight shape. In one embodiment, the vector orientationsare used to distinguish catheter and vessel. Locally, vessels may havesmall regions of orientation similar to the orientation of the catheter.The eigenvectors orientations of the catheter are in general closed to90 degrees, while those of the vessels are not.

In one embodiment, a method of catheter detection is used thatincorporate a Frangi filter for vesselness as well as for shapefeatures. In one embodiment, the method includes determining at onescale only (sigma=4 which reflects the typical width in pixels of thecatheter, as observed in the angiography dataset) the vesselness measureimage and direction image based on the eigenvectors of the Hessianimage. The catheter in a given image frame of angiography data can beisolated using various criteria. These criteria include the direction(threshold of direction image), the length of the connected componentcontaining the catheter (the length of catheter profile should be atleast half of the maximum image dimension in x (or y).

As a constraint for the image processing software, if the catheter isdetected such that it appears in a given image, it is typically the casethat the catheter crosses almost the whole image. In one embodiment, thesystem is programmed to assume the catheter always cuts the bottomboundary of the image. As a result, a lower bound can be set for thesize of the detected object. In addition, once the regions of theangiography image associated with the catheter have been detected, it isuseful to dilate or otherwise expand a boundary by a small incrementaround the centerline of the catheter to ensure a large enough featurehas been detected. An example of a catheter detected based on the stepsoutlined above is shown in FIG. 9E.

Exemplary Catheter Removal Embodiments

As discussed above, the presence of the catheter in the field of viewfor a given angiography image may interfere with various steps andprocessing stages described herein. Accordingly, once the catheter hasbeen detected such as by the software-based methods recited herein, itis desirable to remove the catheter. The bounded area in FIG. 9A showsthe catheter overlapping a blood vessel at an angle. Various objectelimination approaches for removing the catheter while still attemptingto preserve the integrity of the image can be used. Based on a mask ofthe catheter, such as can be generated from or as an output of thecatheter detection process used, a software module can be configured toremove the catheter mask by eliminating the catheter.

One advantageous approach to remove the catheter uses the principle ofsuperpositioning of functions to cancel out and remove when out of phaserelative to each other. In one embodiment, a superposition-basedsoftware module is used to perform catheter removal such as byestimating its intensity profile and reducing it from the image. Acatheter intensity profile can be generated based upon sampling thepoints of the image identified as part of the catheter through acatheter detection software module.

As shown in FIGS. 10A and 10B, an exemplary cylinder 190 is shown withvarious lengthwise slices of thickness T0, T1, and T2 as shown. Thecylinder 190 can be viewed as a model representation of the catheter. Tothe extent the catheter and the cylinder 190 are filled with contrastsolution the intensity changes caused by the contrast solution will begreater in the middle along thickness T0 and then decrease moving awayfrom the center T0 to slice T1 and then further decrease as slice T2 isreached. Thus, as there is less contrast solution at the thinner edgesof the catheter relative to the center of the catheter a profile ofintensity for the catheter can be generated and added to the area of theimage where the catheter was detected to remove the catheter from theimage. An exemplary representation of a related catheter removal methodis shown in FIG. 11A.

Given that the catheter has been detected as described herein, a maskassociated with the pixels in the image that make up the catheter can begenerated such as by using a mask region like that shown in FIG. 9E. Inone embodiment, image intensity is sampled in the catheter region, suchas for example, on lines perpendicular to the catheter line. Theseperpendicular lines span the gradient of contrast solution intensitychanges that gradually decrease from one side of the catheter until alow or relative extremum is reached corresponding to the thickest middleportion of the catheter and then gradually increase again as thecatheter cross-section thins at the edge of the catheter as show inFIGS. 10A and 10B. Each line sampled in the catheter area generates anintensity curve. The various intensity curves can be averaged to asingle curve. This intensity curve can be inverted and then superimposedon the perpendicular lines that make up the catheter region toeffectively remove the catheter from that region as shown in FIG. 11A.

Exemplary Shadow Removal Embodiments

The classical Hessian based filter is a part of the preprocessing and isbased on the eigenvalues of the Hessian of the image. In one embodiment,the Hessian is computed at a number of discrete scales and then themaximum response among them is taken. In one embodiment of a shadowremoval process, scales from 1 to 5 are used. Scale 5 can be chosen asthe scale that best represents the maximum typical observed vessel widthin the available data. Examples of original images and then processed toremove shadows and other features are shown in FIGS. 6I-6N.

The shadow removal preprocessing step is applied in order to transforman original image to a modified image having an improved contrast level.In addition, the modified images is changed by the process of applyingthe Hessian such that it is substantially free of the influence of theheart and diaphragm shadows which can induce several regions or planesof different contrasts. Removing these shadows is desirable because suchregions or planes can lead to incorrect vessel centerlines. In oneembodiment, the shadow removal step includes applying a bottom hatoperator with a filter kernel configured to have a distance parameterthat is much larger than the typical blood vessel width. FIGS. 6L and 6Jshow modified images that been improved by performing a shadow removalprocess.

Exemplary Vessel Centerline (Trace) Generation Embodiments

The two anchor points, distal and proximal, mark the end points andstart point of the vessel centerline. The anchor points are reflected onthe vessel skeleton and the Dijkstra algorithm is applied to find theshortest path in terms of smoothness. FMM is also applied to find ashortest path in terms of intensity (the FMM runs on the enhancedHessian image). Results from the FMM are combined with Dijkstra resultto produce the best vessel centerline (trace) between the two anchorpoints. Vessel centerlines in other angiography frames are generated byapplying conformal mapping combined with FMM to the first generatedtrace.

In one embodiment, the fast marching technique or method deals withefficient computation of geodesic distances based on a potential. In oneembodiment, when the contrast agent is present the potential can be theenhanced Hessian image. In one embodiment, when only the guidewire ispresent (even if visible on the angiography image in a piecewisemanner), such as when no contrast agent is present, the potential isadjusted by constructing a function based on the distance transform. Onemethod for the computation of the potential function the front willpropagate on can be performed by a guidewire-based potential by applyinga Euclidian distance transform on a binary image. Once the distancetransform is generated such a transform can be further modified into apotential function by applying an exponent to a negative fractionalpower times the distance transform. An exemplary guidewire potential isshown in FIG. 6G.

FIG. 5B shows a process flow 170 relating to vessel centerlinegeneration. In one embodiment, a Hessian having a scale of 1 is appliedto a frame of angiography data 170 a. This application of the Hessianresults in enhancements of thin ridges in the image such as theguidewire. In one embodiment, automatic detection of the guidewire andselection of an anchor point on the guidewire is performed 170 c. Oncethe guidewire is detected, in one embodiment, the point with the highestLoG response is identified as the anchor point. Tracking distalguidewire anchor point to all pullback angiography frames 170 e isperformed next. The proximal anchor point is detected in a single frame.The distal anchor point is also detected in a single frame. In oneembodiment, each anchor point is a feature that can be easily detectedin other frames by means of tracking. Next, anchor points are tracked toall frames so that each angiography frame will have two end points forvessel-centerline generation (trace).

In one embodiment, a user selected point such as a guidewire point on anangiography image is selected 170 j. In turn, a Hessian of scale (up toabout 5) can be applied 170 k to an angiography image in order toenhance the vessels. The modified image as a result of the applicationof the Hessian can then be used to perform detection of nearestbifurcation anchor point 170 l. This detection step can use the userselected point or hint point as an input. Next, detection of theextrapolation of an anchor point is performed 170 m. Please clarifywhich anchor point is being detected. Next, tracking anchor points toall pullback angiography is performed frames 170 n.

In one embodiment, the system next uses a graph search software module,such as a Dijkstra shortest path solution for a graph. Applying theDijkstra algorithm or other shortest path algorithm combined with FMMand selecting a best initial vessel centerline can then be performed 170f with regard to the angiography pullback frames. Tracking the vesselcenterline in angiography pullback frames using FMM on a narrow bandbased on conformal mapping is then performed 170 g. In this context,narrow band means building a narrow band region around the trace ofinterest. This narrow band is intended to increase the efficiency of theFMM algorithm, due to computation of geodesic distances on a restrictedregion of the image. These centerlines can be stored in one or moretables and displayed on the applicable angiography images.

Exemplary Marker Detection and Co-Registration Embodiments

FIG. 5C shows a process flow 180 relating to marker detection andco-registration. As used herein, the term trace can be interchangeablewith centerline. Initially, as an input the vessel centerlines (traces)from the pullback frames are provided as an input for samplingorientation 180 a. In addition, a LoG is applied to images from thepullback frames 180 c. Sampling LoG images perpendicular to the tracesis performed 180 e. In one embodiment, dynamic programming is performedor iterations are run with different starting or ending points to findmarker positions in all frames 180 g. In one embodiment, the dynamicprogramming or iterative process can be implemented using the Viterbialgorithm. Next, a selection of the most probable solution for a markeron a per frame basis is performed 180 h. Calculation of marker positionalong with the marker normalized arc-length position along the vesselcenterline in all frames is performed 180 l.

Next, co-registering of all combinations of OCT and angiography framesbased on calculated marker position in terms of arc length can beperformed. Since all vessel centerlines start and end at the sameanatomical features in all angiography frames, each centerlinecorresponds to the other centerlines in other frames. Therefore,centerline length or arc-length can be used as a basis forco-registration. The marker position in terms of arc length is preserved(up to some error) in all frames.

One challenge encountered when attempting to resolve the opaque markerbands of a sensor or data collection probe is the use of contrastsolution as part of an OCT pullback. In one embodiment, it is useful toprocess frames of angiographic data prior to the introduction ofcontrast solution so that the guidewire and imaging catheter can be usedto provide an initial path through the blood vessel. This initialdataset can be iteratively improved upon using other information andparameters as described herein.

A Viterbi based algorithm automatically detects the radiopaque marker ineach image of the pullback. This algorithm can be used to obtain globalsolution based on blob intensity and predication of location (constantvelocity along trace). As a prerequisite for this algorithm, a processof detecting and tracking the vessel centerlines (traces) is performed.The traces are used to create a continuous co-registration between theOCT and angiography. These curves are computed by means of the fastmarching method. The fast marching method allows, on each frame,efficient computation of paths (traces) between the proximal point(which can be the user selected point or hint point) and the distalstationary marker. The stationary marker is detected on a frame (withand/or without contrast agent/dye). Template matching technique isemployed to track both the proximal point and the distal marker over thesubsequent sequence.

The Viterbi algorithm is configured to balance an extrinsic factor andan intrinsic factor. The extrinsic factor (marker band indications) isderived from the marker band Laplacian of the Gaussian map by resamplingthe map in discrete strips perpendicular to the trace, per angiographyframe. The intrinsic factor is the arc-length progression over time.This intrinsic factor models the advancement of the marker band alongthe pullback's arc length. The basic notion is that the average pace isdetermined by the pullback speed, while there are penalties fordeviating from this pace. This factor takes the natural “sawing” profileinto account, by penalizing forward/backward motion differently.

FIG. 12 shows a data collection and co-registration system 300 thatincludes various software and hardware components suitable forprocessing intravascular and angiographic image data. In one embodiment,once one or more frames of OCT image data and angiography image data areco-registered the output is a registration table. In one embodiment,frame of OCT data can be monitored to check for clear frame indicationstate and this clear frame indication can be used to trigger the Cinesuch that the frames of angiography data can be captured. In oneembodiment, for a given pullback procedure during which a probe is pullbacked through a blood vessel while probe data and angiography data arecollected, time stamping of frames, registration table population, andimage processing features, and other processes may be performed.

The user interface (UI) 308 is in communication with the OCT adaptor320. The Image Processing Module 330 is in communication with the OCTadaptor 320. In one embodiment, the image processing module 330 performsor applies operators or transforms to frames of angiography data such asfor shadow removal, guidewire detection, catheter removal, and otherimage processing steps outlined herein. The optical coherence tomographysystem 310 is in communication with the OCT adaptor 320. The opticalcoherence tomography system 310 can include or be in communication withthe framer grabber 302. Angiography frames are grabbed using the framegrabber and fetched by the software module.

The OCT frame table 315 includes information and images of a bloodvessel obtained during a pullback of an imaging probe through the bloodvessel. The role of the OCT adaptor 320 is provide a software interfacebetween the angiography system and the OCT system.

The software-based systems, such as the server or workstation describedherein, and the software modules configured to automatically run andcapture the angiography images and tag each image by its acquisitiontime support co-registration of intravascular data tagged with anacquisition time. The image processing module 330 which can include aco-registration software module automatically detects the radio-opaquemarker on each angiography image corresponding to the intravascularacquisition. A single user input may be requested to assist with thedetection as shown in FIG. 5B. The co-registration software modulecomputes the intravascular imaging catheter's path on all angiographyimages corresponding to the intravascular image acquisition during thepullback of the probe through the vessel being imaged. Theco-registration software module produces a co-registration table of theacquisition's intravascular and external images that include theradio-opaque marker's location on each angiography image; position ofeach intravascular image/data point on each angiography image; and a FOMassociated with each co-registration result, providing a measure of thelevel of confidence in the veracity of that result.

The user is presented with graphic representations of the intravascularand angiographic images, and with the correspondence between the two,such as location of a certain intravascular image on the angiographicimage as part of a user interface when co-registration is complete inone embodiment. If during a co-registration procedure a FOM orconfidence score is not acceptable, additional user input or otherparameters from the OCT system may be requested or automaticallyobtained.

Exemplary Confidence Score/Figure of Merit Embodiments

For each detection of a probe marker a confidence score also referred toas a FOM assigned to each detected probe marker. Score is based on oneor more of blob intensity, the number of dark blobs in the vicinity ofthe predicted area of the marker, the marker arc-length along thetraces, the blob movement, and the stability of traces. The FOM/Scorereflects a confidence measure in the returned results. In oneembodiment, the score is in the range [0, 1] where 0 reflects the lowestconfidence and 1 reflects the highest.

The angiography related software modules, such as one or more modulesdescribed herein, are evaluating images generated using imaging devicesthat are typically disposed outside of a subject's body. In contrast, adata collection probe, such as an OCT, IVUS, FFR, pressure, or otherdata collection modality, can be disposed within the blood vessel of apatient. As a result, data obtained from such a data collection probeduring a pullback or previously known as a data collection proberelating parameter can be used by the angiography software to improvethe operation of the methods and stages described herein. An adaptersoftware module or other software module can be used to provide OCTinformation to the angiography image frame processing software modulesand vice versa.

For example, the following parameters relating to data obtained withregard to a blood vessel as part of the intravascular data collection,can be transmitted to the angiography software or other software modulesfor analysis or to otherwise help evaluate the subject or otherwiserelate different datasets, pullback length in mm, start of pullback, endof pullback, indications of bifurcations such as side branches fromcollected OCT data, data collected with regard to frames prior to theintroduction of a contrast agent or dye, OCT and angiographysynchronized frames time-tags, pullback velocity, distance between thedistal and the proximal markers of the catheter and other factors andparameters obtained with respect to a given data collection modalitysuch as longitudinal blood vessel image data, pressure data, EKG data,systole state during pullback, diastole state during pullback, and otherinformation available relating to a subject.

Angiography Table

The angiography table, such as shown in FIG. 14, contains informationthat describes the angiography pullback as well as each angiographyframe acquired. The angiography table is created by the angiographysoftware module at acquisition and is partially populated with timestamp data. This table is extracted by the OCT module at the completionof acquisition and stored. The table is then provided to the angiographysoftware module at co-registration time, when the co-registrationdependent fields are populated.

Co-Registration Table

The co-registration table contains the results of a successfulco-registration as shown in FIG. 15. It contains all of theOCT/angiography cross-reference information necessary to drive theco-registration GUI toolset. This table contains an entry for each OCTframe which contains that frame's acquisition time stamp and a list withan entry for each angiography frame containing the OCT marker positioninformation. In one embodiment, the co-registration table associates theOCT frame index with the registered angiography frame index.Additionally, the table can include an entry which associates an OCTframe and angiography frame.

Additional Multimodal Co-Registration Features and Embodiments

In one embodiment, co-registration refers to synchronizing frames fromtwo or more data collection modalities or otherwise combininginformation from two or more data collection modalities. For example,bifurcations detected on OCT images can be used as anchors with respectto bifurcations detected on angiography images. The co-registrationfeatures recited here are not limited to OCT. Instead, the featuresdescribed here relating to co-registering imaging or other datacollection modalities relating to the vascular system and individualblood vessels can be extended to other intravascular imaging modalities.In one embodiment of the invention, the centerline of the vessel isdetermined from the path of a guidewire or catheter that is tracked by atracking system such as the Medical Position System of Mediguide duringits advancement through the vessel.

In one embodiment, side branches detected in OCT frames of data usingOCT image processing can be used as an input to improve co-registrationwith angiography data. For example, in one embodiment, each OCT frameincludes a flag (yes/no) indicating if a sidebranch is present. Further,once the co-registration is obtained, positions of stents, calciumdeposits, lipid deposits, thrombus, thin capped fibroatheromas (TCFAs or“vulnerable plaques”), vessel normalization, side branch detection, FFRvalues (which may be computed as a vascular resistance ratio (VRR)values based on OCT image data), lumen size values, stents, and variousother data described herein can be overlaid on the angiography image orthe OCT images in light of the co-registration between the datasets.

Live Stent Placement Guidance Embodiments and Features

In one embodiment, following OCT/angiography co-registration, theguidewire is retained post-pullback for stent placement via anothercatheter. The process of imaging the blood vessel that was the subjectof the pullback continues via continued fluoroscopic imaging,co-registered to OCT. Moving along the OCT frames or angiography framesallows side branches and other information to be seen. In oneembodiment, various processing steps are performed with regard to theOCT data such as detection of prior stents, 3-D co-registered virtualhistology, lumen detection, guidewire detection, stent malapposition,plaque detection, and others. Since the OCT and angiography frames areregistered, information found in the OCT frames can be overlaid on theangiography screen that the operator will use to place a stent. If sidebranches can be shown in the angiography view on a user interface, thiscan help avoid the unwanted caging of a side branch during stentdeployment.

In addition, various types of overlays relating to stents that have beenpreviously deployed or that are candidates for deployment can bedisplayed on one or both of an OCT image and angiography image followingco-registration. For example, bioresorbable vascular scafford (BVS), anew type of stent that is radio-translucent, can be detected on OCTframes using OCT image processing. This OCT image data can be used toprovide a specific type of stent overlay that is important in thecontext of the angiography data because such stents are not made visibleby x-rays. As another special case of data overlay, in one embodiment,stent malapposition information from a given OCT frame can be used tocolor code or generate other indicia to modify the stent image on theX-ray image to show regions of malapposition.

In addition, given that a marker on the stent delivery probe can betracked, a stimulated stent can be shown in relation to the marker onthe OCT longitudinal mode or L-mode. The angiography/OCT co-registrationallows cross-correlating of tissue features, lumen features and movingfeatures such as a balloon or stent insertion to be shown in theangiography with overlays and with the display of elements such as astent cross-section in the L-mode. If a scan of the stent is obtained asa wireframe model or is selected from a drop down menu prior tostenting, the diameter and length can be used to display the stent onthe L-mode or angiography with greater accuracy.

In one embodiment, bands on the OCT image and/or the angiography imageshowing regions to avoid stenting like side branches and a targetdeployment region based on stenosis/MLA calculations can be used. Theangiography and OCT displays can be used to show a greater level ofgranularity with overlays to help a user properly position a stentwithin a target area. In addition, given the wireframe model of thestent and the calculated lumen areas from the OCT frames that areco-registered with the location of the stent on the angiography system,visual guidance for a stent inflation target can be provided anddisplayed. In one embodiment, this can be performed using a simulatedwireframe of the stent and the expanding balloon used to selectivelyexpand one or both ends of the stent. These types of investigationsusing OCT and angiography can be used on a pre-stent, post-stent, or aspart of future follow ups.

In one embodiment, when a pullback is performed the OCT data andangiography data are stored. This stored data can be used to generateimages or a model of an artery. With such a model, live stent placementduring a subsequent pullback is enhanced. In this way, the priorexisting OCT/angiography co-registration information can be used as abaseline.

Angiography data can also be used to inform or improve or correct OCTimage display features or detection algorithms. One correction in OCTfrom angiography data is to re-space the OCT frames on the L-mode toshow the actual, physical separation between frames as measured by theco-registration tool. This compensates for spacing errors that arisefrom assuming a constant pullback velocity relative to the vessel lumen.In reality the pullback velocity varies significantly due to cardiacmotion and our frames are not equally spaced. A software module can beused to measure the frame-to-frame spacing accurately once co-registeredOCT and angiography datasets are generated. A per frame correction canbe applied to re-space the L-mode view in a given user interface. Thiscan also be applied to 3D OCT renderings, which would provide a moreaccurate visual representation of the vessel.

In general, by having a co-registered set of frames and bidirectionalcommunication between angiography and OCT systems, various additionalbenefits are possible. The angiography information includes traces thathave been generated for different vessels. The junctions of thesebranches can be mapped to particular frames to inform OCT side branchdetection. In one embodiment, by storing angiography data and OCTobtained during angiography, a record can be built over time that can beused to co-register OCT images at different time points with theangiography data acting as a bridge or linker between two different OCTdatasets.

In one embodiment, a pressure probe or other data collection modalitycan be used to collect data to improve the representation of a bloodvessel using another imaging modality or parameters. In one embodiment,VRR can be used to calculate the percentage contribution of eachstenosis to an overall FFR value and display the relative percentages onthe angiography image. In addition, side branch position informationfrom the OCT images or the angiography images can be used to improve VRRcalculation by identifying additional junctions and points of flow inthe area near a blood vessel being imaged.

In one embodiment, the system and methods can be used to monitor athrombectomy catheter in OCT L-mode: This can be used with guidedstenting using simulated stents and registration data as describedherein. In general, in part the invention relates to the tracking of anytherapeutic device with a radio-opaque marker band, and displaying itsposition on the OCT L-mode and the previously-acquired co-registeredX-ray images. The therapeutic device can be a stent or a thrombectomycatheter, or a balloon device such as an angioplasty balloon ordug-eluting balloon, or an excisional device such as a rotationalatherectomy probe (Rotablator).

Non-Limiting Software Features and Embodiments for ImplementingAngiography and Intravascular Data Collection Methods and Systems

The following description is intended to provide an overview of devicehardware and other operating components suitable for performing themethods of the invention described herein. This description is notintended to limit the applicable environments or the scope of theinvention. Similarly, the hardware and other operating components may besuitable as part of the apparatuses described above. The invention canbe practiced with other system configurations, including personalcomputers, multiprocessor systems, microprocessor-based or programmableelectronic devices, network PCs, minicomputers, mainframe computers, andthe like.

Some portions of the detailed description are presented in terms ofalgorithms and symbolic representations of operations on data bitswithin a computer memory. These algorithmic descriptions andrepresentations can be used by those skilled in the computer andsoftware related fields. In one embodiment, an algorithm is here, andgenerally, conceived to be a self-consistent sequence of operationsleading to a desired result. The operations performed as methods stopsor otherwise described herein are those requiring physical manipulationsof physical quantities. Usually, though not necessarily, thesequantities take the form of electrical or magnetic signals capable ofbeing stored, transferred, combined, transformed, compared, andotherwise manipulated.

Unless specifically stated otherwise as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “comparing” or “arc length measuring” or “detecting” or“tracing” or “masking” or “sampling” or “operating” or “generating” or“determining” or “displaying” or the like, refer to the action andprocesses of a computer system, or similar electronic computing device,that manipulates and transforms data represented as physical(electronic) quantities within the computer system's registers andmemories into other data similarly represented as physical quantitieswithin the computer system memories or registers or other suchinformation storage, transmission or display devices.

The present invention, in some embodiments, also relates to theapparatus for performing the operations herein. This apparatus may bespecially constructed for the required purposes, or it may comprise ageneral purpose computer selectively activated or reconfigured by acomputer program stored in the computer.

The algorithms and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct more specializedapparatus to perform the required method steps. The required structurefor a variety of these systems will appear from the description below.

Embodiments of the invention may be implemented in many different forms,including, but in no way limited to, computer program logic for use witha processor (e.g., a microprocessor, microcontroller, digital signalprocessor, or general purpose computer), programmable logic for use witha programmable logic device, (e.g., a Field Programmable Gate Array(FPGA) or other PLD), discrete components, integrated circuitry (e.g.,an Application Specific Integrated Circuit (ASIC)), or any other meansincluding any combination thereof. In a typical embodiment of thepresent invention, some or all of the processing of the data collectedusing an OCT probe, an FFR probe, an angiography system, and otherimaging and subject monitoring devices and the processor-based system isimplemented as a set of computer program instructions that is convertedinto a computer executable form, stored as such in a computer readablemedium, and executed by a microprocessor under the control of anoperating system. Thus, user interface instructions and triggers basedupon the completion of a pullback or a co-registration request, forexample, are transformed into processor understandable instructionssuitable for generating OCT data, performing image procession usingvarious and other features and embodiments described above.

Computer program logic implementing all or part of the functionalitypreviously described herein may be embodied in various forms, including,but in no way limited to, a source code form, a computer executableform, and various intermediate forms (e.g., forms generated by anassembler, compiler, linker, or locator). Source code may include aseries of computer program instructions implemented in any of variousprogramming languages (e.g., an object code, an assembly language, or ahigh-level language such as Fortran, C, C++, JAVA, or HTML) for use withvarious operating systems or operating environments. The source code maydefine and use various data structures and communication messages. Thesource code may be in a computer executable form (e.g., via aninterpreter), or the source code may be converted (e.g., via atranslator, assembler, or compiler) into a computer executable form.

The computer program may be fixed in any form (e.g., source code form,computer executable form, or an intermediate form) either permanently ortransitorily in a tangible storage medium, such as a semiconductormemory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-ProgrammableRAM), a magnetic memory device (e.g., a diskette or fixed disk), anoptical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card),or other memory device. The computer program may be fixed in any form ina signal that is transmittable to a computer using any of variouscommunication technologies, including, but in no way limited to, analogtechnologies, digital technologies, optical technologies, wirelesstechnologies (e.g., Bluetooth), networking technologies, andinternetworking technologies. The computer program may be distributed inany form as a removable storage medium with accompanying printed orelectronic documentation (e.g., shrink-wrapped software), preloaded witha computer system (e.g., on system ROM or fixed disk), or distributedfrom a server or electronic bulletin board over the communication system(e.g., the internet or World Wide Web).

Hardware logic (including programmable logic for use with a programmablelogic device) implementing all or part of the functionality previouslydescribed herein may be designed using traditional manual methods, ormay be designed, captured, simulated, or documented electronically usingvarious tools, such as Computer Aided Design (CAD), a hardwaredescription language (e.g., VHDL or AHDL), or a PLD programming language(e.g., PALASM, ABEL, or CUPL).

Programmable logic may be fixed either permanently or transitorily in atangible storage medium, such as a semiconductor memory device (e.g., aRAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memorydevice (e.g., a diskette or fixed disk), an optical memory device (e.g.,a CD-ROM), or other memory device. The programmable logic may be fixedin a signal that is transmittable to a computer using any of variouscommunication technologies, including, but in no way limited to, analogtechnologies, digital technologies, optical technologies, wirelesstechnologies (e.g., Bluetooth), networking technologies, andinternetworking technologies. The programmable logic may be distributedas a removable storage medium with accompanying printed or electronicdocumentation (e.g., shrink-wrapped software), preloaded with a computersystem (e.g., on system ROM or fixed disk), or distributed from a serveror electronic bulletin board over the communication system (e.g., theinternet or World Wide Web).

Various examples of suitable processing modules are discussed below inmore detail. As used herein a module refers to software, hardware, orfirmware suitable for performing a specific data processing or datatransmission task. In one embodiment, a module refers to a softwareroutine, program, or other memory resident application suitable forreceiving, transforming, routing and processing instructions, or varioustypes of data such as angiography data, OCT data, FFR data, IVUS data,co-registration table data, centerlines, shadows, pixels, intensitypatterns, and other information of interest as described herein.

Computers and computer systems described herein may include operativelyassociated computer-readable media such as memory for storing softwareapplications used in obtaining, processing, storing and/or communicatingdata. It can be appreciated that such memory can be internal, external,remote or local with respect to its operatively associated computer orcomputer system.

Memory may also include any means for storing software or otherinstructions including, for example and without limitation, a hard disk,an optical disk, floppy disk, DVD (digital versatile disc), CD (compactdisc), memory stick, flash memory, ROM (read only memory), RAM (randomaccess memory), DRAM (dynamic random access memory), PROM (programmableROM), EEPROM (extended erasable PROM), and/or other likecomputer-readable media.

In general, computer-readable memory media applied in association withembodiments of the invention described herein may include any memorymedium capable of storing instructions executed by a programmableapparatus. Where applicable, method steps described herein may beembodied or executed as instructions stored on a computer-readablememory medium or memory media. These instructions may be softwareembodied in various programming languages such as C++, C, Java, and/or avariety of other kinds of software programming languages that may beapplied to create instructions in accordance with embodiments of theinvention.

The aspects, embodiments, features, and examples of the invention are tobe considered illustrative in all respects and are not intended to limitthe invention, the scope of which is defined only by the claims. Otherembodiments, modifications, and usages will be apparent to those skilledin the art without departing from the spirit and scope of the claimedinvention.

The use of headings and sections in the application is not meant tolimit the invention; each section can apply to any aspect, embodiment,or feature of the invention.

Throughout the application, where compositions are described as having,including, or comprising specific components, or where processes aredescribed as having, including or comprising specific process steps, itis contemplated that compositions of the present teachings also consistessentially of, or consist of, the recited components, and that theprocesses of the present teachings also consist essentially of, orconsist of, the recited process steps.

In the application, where an element or component is said to be includedin and/or selected from a list of recited elements or components, itshould be understood that the element or component can be any one of therecited elements or components and can be selected from a groupconsisting of two or more of the recited elements or components.Further, it should be understood that elements and/or features of acomposition, an apparatus, or a method described herein can be combinedin a variety of ways without departing from the spirit and scope of thepresent teachings, whether explicit or implicit herein.

The use of the terms “include,” “includes,” “including,” “have,” “has,”or “having” should be generally understood as open-ended andnon-limiting unless specifically stated otherwise.

The use of the singular herein includes the plural (and vice versa)unless specifically stated otherwise. Moreover, the singular forms “a,”“an,” and “the” include plural forms unless the context clearly dictatesotherwise. In addition, where the use of the term “about” is before aquantitative value, the present teachings also include the specificquantitative value itself, unless specifically stated otherwise.

It should be understood that the order of steps or order for performingcertain actions is immaterial so long as the present teachings remainoperable. Moreover, two or more steps or actions may be conductedsimultaneously.

Where a range or list of values is provided, each intervening valuebetween the upper and lower limits of that range or list of values isindividually contemplated and is encompassed within the invention as ifeach value were specifically enumerated herein. In addition, smallerranges between and including the upper and lower limits of a given rangeare contemplated and encompassed within the invention. The listing ofexemplary values or ranges is not a disclaimer of other values or rangesbetween and including the upper and lower limits of a given range.

It should be appreciated that various aspects of the claimed inventionare directed to subsets and substeps of the techniques disclosed herein.Further, the terms and expressions employed herein are used as terms ofdescription and not of limitation, and there is no intention, in the useof such terms and expressions, of excluding any equivalents of thefeatures shown and described or portions thereof, but it is recognizedthat various modifications are possible within the scope of theinvention claimed. Accordingly, what is desired to be secured by LettersPatent is the invention as defined and differentiated in the followingclaims, including all equivalents.

What is claimed is:
 1. A processor-based method of displaying anangiographic and an intravascular representation of a blood vesselcomprising: generating a set of optical coherence tomography image datain response to distance measurements of the blood vessel obtained duringa pullback of a probe through the blood vessel using an opticalcoherence tomography system, the set of optical coherence tomographyimage data comprising a plurality of cross-sectional image at aplurality of positions along the blood vessel; generating a set ofangiography image data using an angiography system during the pullbackof the probe through the blood vessel using an optical coherencetomography system, the set of angiography image data comprising aplurality of two-dimensional images obtained at different points in timeduring the pullback; displaying, on a graphical user interface, a firstpanel comprising a first longitudinal view of the blood vessel generatedusing the optical coherence tomography image data; and displaying, on agraphical user interface, a second panel comprising a frame of theangiography image data identifying the blood vessel using one or morepoints in the frame and a vessel centerline passing through the one ormore points.
 2. The method of claim 1, further comprising co-registeringthe optical coherence tomography image data and the angiography datausing vessel centerlines to create a continuous registration of atracked marker, wherein the tracked marker is disposed on an opticalcoherence tomography data collection probe.
 3. The method of claim 1further comprising co-registering the optical coherence tomography imagedata and the angiography data such that selecting a point along thevessel centerline through a user interface changes a frame identifier inthe first longitudinal view.
 4. The method of claim 2 further comprisingusing pullback speed or pullback length to perform an iterative searchto reject candidates for the tracked marker based on the possiblelocations for such markers based upon the pullback length and/orpullback speed.
 5. The method of claim 1 wherein the vessel centerlineis generated using a shortest path technique and a plurality ofprocessing steps from a Dijkstra algorithm.
 6. The method of claim 1further comprising the step of removing a guide catheter image from oneor more frames of angiography data using superposition of an intensityprofile.
 7. The method of claim 1 wherein the vessel centerline isgenerated using path information generated from one or more angiographyframes in the absence of contrast solution.
 8. The method of claim 1further comprising generating a confidence score for each detection andco-registration between angiography data and optical coherencetomography data.